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Introduction 

Developing  and  effectively  managing  schedules  is  critical  to  the  success  of  a 
program.  Program  managers  are  becoming  increasingly  aware  of  the  need  for  greater 
accuracy  in  schedule  estimation,  assessment,  and  risk  management  to  control  cost  and 
deliver  on  time.  A  Government  Accountability  Office  (GAO)  assessment  of  86  programs  that 
made  up  the  2012  portfolio  of  Major  Defense  Acquisition  Programs  (MDAPs)  found  that  the 
portfolio  experienced  total  acquisition  cost  growth  of  38%.  In  addition,  the  average  schedule 
delay  in  delivering  initial  capability  was  27  months  when  measured  against  first  full 
estimates  (GAO,  2013),  representing  a  69%  increase  over  a  12-year  period.1  Most  Major 
Automated  Information  Systems  (MAIS)  programs  experienced  schedule  delays  ranging 
from  six  months  to  10  years.  Clearly,  schedule  can  pose  a  significant  risk  and  drive  cost 
growth. 

The  purpose  of  this  research  was  to  help  strengthen  the  acquisition  community’s 
ability  to  produce  data-driven  realism  in  program  schedules.  This  research  effort  had  three 
main  focus  areas:  (1)  compile  schedule  data  from  programs  to  identify  key  schedule  drivers 
and  characteristics  and  build  a  data  repository,  (2)  analyze  the  data  from  statistical  and 
qualitative  perspectives,  and  (3)  document  data  collected  and  analysis  performed,  and  how 
it  can  be  accessed  for  analysis. 

The  detailed  approach  used  in  the  research  was  comprised  of  the  following  high- 
level  steps: 


•  Identify  and  review  primary  data  sources 


1  Calculated  based  on  GAO  data  that  the  average  schedule  delay  in  delivering  initial  capabilities  was 
16  months  in  2000  and,  compared  to  21  months  in  2007,  represents  a  31%  increase  in  schedule 
delays  over  a  seven-year  period.  See  the  2008  GAO  report  on  95  weapons  systems  cited  by  Meier 
(2010). 
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•  Develop  a  list  of  program  attributes  to  evaluate 

•  Develop  an  Excel-based  data  collection  framework 

•  Collect  data  and  populate  data  repository 

•  Synthesize  and  cleanse  data 

•  Analyze  and  assess  data 

•  Develop  findings 

•  Document  research 

The  initial  focus  of  the  data  collection  phase  was  to  identify,  understand,  and  review 
existing  external  data  repositories  or  sources  that  collect  enterprise-level  acquisition 
information  and  data,  particularly  related  to  program  schedule.  Across  the  federal 
government,  the  Department  of  Defense  (DoD)  sources  and  the  Office  of  Management  and 
Budget  (OMB)  Information  Technology  (IT)  Dashboard  were  data  sources  identified  and 
reviewed  for  this  research. 

Based  on  the  data  source  review,  the  study  team  determined  that  the  Defense 
Acquisition  Management  Information  Retrieval  (DAMIR)  was  the  best  source  of  data  for  DoD 
large  scale  programs  as  it  contained  both  MDAP  and  MAIS  program  data  (DoD,  2015). 
Although  the  DAMIR  data  is  high  level,  it  provided  sufficient  schedule  fidelity  and  additional 
cost  and  program  parameters  of  interest  to  be  useful  for  the  research.  However,  detailed 
reviews  of  the  OMB  IT  Dashboard  data  revealed  that  schedule  data  is  highly  aggregated. 
Program  start  date  and  program  end  date  are  the  only  schedule  parameters  included  with 
no  intermediate  schedule  milestone  data  available  in  the  repository.  The  OMB  IT  Dashboard 
has  a  large  amount  of  data;  however,  without  further  fidelity  and  definition  for  schedule  data, 
it  was  determined  that  it  would  not  be  useful  to  this  research.  No  additional  enterprise-wide 
acquisition  data  sources  with  relevant  schedule  data  were  identified;  however,  there  are 
likely  other  centralized  systems  that  contain  acquisition  information  but  are  not  open  source. 

Ground  rules  and  assumptions  (GR&As)  were  developed  to  bound  and  scope  the 
research  and  establish  baseline  conditions  for  the  analysis.  Key  GR&As  included  the 
following: 

1 .  Only  actual  data  was  analyzed.  Future  schedule  dates  were  excluded  from 
the  analysis. 

2.  Milestone  equivalents  were  assumed  to  compare  data  between  older 
program  milestones  and  new  program  milestones,  e.g.,  MS  II  =  MS  B. 

Programs  with  negative  or  zero  schedule  durations  between  milestones  were 
removed  from  the  analysis  as  these  reflect  acquisition  process  anomalies.  Negative  duration 
values  could  occur  either  when  the  program  had  unique  circumstances  that  caused 
milestones  to  take  place  in  non-sequential  order  or  could  represent  an  error  in  the  data. 

The  study  team  developed  an  Excel-based  data  framework  that  included  schedule 
parameters,  cost  parameters,  and  program  attribute  parameters  that  were  both  available 
and  of  interest  to  analyze.  Cost  by  appropriation,  major  schedule  milestone  dates,  and  life 
cycle  phase  were  collected.  Program  attributes  collected  included  program  type,  assigned 
component,  acquisition  category,  Joint  Capability  Area  (JCA),  new  start  or  modification,  and 
whether  a  breach  occurred  within  the  program.  Additionally,  program  baseline  costs  were 
synchronized  into  constant  year  15  millions  of  dollars  (CY15$M)  to  avoid  any  dollar 
distortions  in  the  data.  Schedule  durations  between  milestones  were  calculated  (in  months) 
and  included  in  the  repository.  The  analysis  focused  on  analyzing  predictors  of  schedule 
duration. 
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In  total,  the  study  team  analyzed  more  than  2,600  data  points,  including  560 
schedule  milestone  dates  from  143  MDAP  and  MAIS  programs  from  DAMIR.  Table  1  shows 
a  summary  of  the  data  points  collected.  Within  the  schedule  data  points,  the  largest  data  for 
MDAP  and  MAIS  programs  were  for  MS  B  and  MS  C.  The  data  had  all  Services 
represented,  with  Navy  and  Air  Force  having  slightly  more  data  points.  Lastly,  the  data 
showed  each  of  the  JCA  areas  represented,  with  Force  Application  having  slightly  more 
data  points.  Although  the  data  was  high  level,  it  provided  sufficient  schedule  fidelity  and 
additional  cost  and  program  parameters  of  interest  to  have  value  for  the  research. 

Table  1.  Data  Collection  Summary 


The  data  analysis  began  with  synthesizing  and  cleansing  the  collected  data.  The 
next  phase  was  characterizing  the  data.  The  size  and  makeup  of  the  data  by  the  different 
parameters  collected  was  analyzed  for  insights,  trends,  and  relationships.  The  analytic  tool 
was  Excel  with  data  analysis  add-in  features  as  well  as  graphing  capability.  Scatter  plots, 
trend  lines,  and  various  statistical  analysis  were  conducted  to  gain  insight  into  what 
relationships  the  data  may  reveal.  Additional  ways  of  analyzing  the  data,  such  as  changes 
over  time,  were  also  performed  for  further  insight. 

Among  the  key  findings  was  a  wide  range  of  variability  and  a  lack  of  strong  linear 
relationships  between  schedule  durations  and  program  attributes  analyzed.  This  implies  that 
the  complexity  of  DoD  large-scale  programs  was  not  easily  explained  by  predictive 
parameters  such  as  cost,  program  type,  JCA,  and  Service,  for  example.  Despite  this,  the 
research  revealed  several  emergent  trends.  Presented  in  this  summary  are  two  emergent 
trends: 


Trend  1 :  Average  MS  B  to  MS  C  durations,  which  accounts  for  the  Engineering  and 
Manufacturing  Development  (EMD)  Phase  of  the  DoD  acquisition  life  cycle,  have  decreased 
43%  for  MAIS  and  42%  for  MDAPs  over  the  last  25  years,  as  shown  in  Figures  1  and  2.  This 
reduction  in  the  EMD  phase  average  duration  was  not  easily  explained  by  the  data,  but 
suggests  that  various  efforts  to  improve  acquisition  outcomes  may  have  contributed  to 
reduced  development  schedules.  For  example,  in  the  last  25  years,  large  acquisition 
programs  have  trended  away  from  single  pass  (aka,  “Big  Bang”)  efforts  in  favor  of 
incremental  development  and  delivery  of  needed  capabilities. 
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Figure  1.  MDAP  Development  Times 
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Figure  2,  MAIS  Development  Times 

Trend  2:  From  a  cost  correlation  perspective,  the  study  team  observed  that  higher 
Research,  Development,  Test,  and  Evaluation  (RDT&E)  costs  as  a  percentage  of  total 
Acquisition  Costs  correlated  with  shorter  EMD  schedule  durations  for  both  MAIS  and  MDAP 
efforts.  This  observation  suggests  that  development  schedules  may  be  “bought  down”  with  a 
greater  share  of  the  total  acquisition  budget,  as  depicted  in  Figures  3  and  4. 
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Figure  3.  MDAP  RDT&E  Percentage  vs.  Schedule  (MS  B  to  MS  C) 


Figure  4.  MAIS  RDT&E  Percentage  vs.  Schedule  (MS  B  to  MS  C) 

The  data  collected  during  this  research  can  be  used  to  help  validate  schedule 
realism  and  identify  schedule  outliers  compared  to  major  DoD  programs  with  similar 
program  attributes.  For  more  comprehensive  analysis  and  higher  confidence  predictive 
measures,  additional  parameter  analysis  such  as  requirements  and  funding  stability, 
program  office  maturity,  governance  structures,  and  technology  maturity  is  needed.  Also,  as 
noted,  the  dataset  covered  only  large  DoD  programs.  The  study  team  did  not  find  useful 
data  for  civil-sector  and  below  ACAT-I  threshold  DoD  programs. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-93- 


In  summary,  the  data  suggested  some  emergent  trends;  however,  the  data  had  a  lot 
of  variability  and  a  wide  range  for  most  schedule  milestones.  The  data  also  does  not  appear 
to  have  strong  linear  relationships  between  schedule  milestone  length  and  program 
attributes  analyzed  for  either  MAIS  or  MDAPs.  One  explanation  is  that  the  complexity  of 
large-scale  DoD  programs  is  not  easily  explained  in  full  by  the  selected  predictive 
parameters.  Additional  parameter  analysis  is  needed  to  explain  and  predict  schedule 
differences  (such  as  funding  stability,  program  manager  experience,  requirements  change, 
technology  readiness  levels,  etc.),  but  these  parameters  may  be  difficult  to  collect.  However, 
the  data  collected  is  a  rich  dataset  that  can  be  used  for  analogy  comparisons  for  large  DoD 
programs.  The  study  team  recommends  the  data  be  used  to  perform  high-level  schedule 
assessments  or  for  analogies  to  evaluate  schedule  realism  throughout  a  program’s  life 
cycle. 
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Context 


I  2  | 


Program  schedules  are  often  driven  by 

Operational  commanders  expectations 

-  Acquisition  Executive’s  direction 

-  Budget  availability  and  type 

-  PMO  guestimates 

-  Politics 


Instead  of  data  driven  analytics  of  related  programs 
Leading  to 

-  Delays  against  unrealistic  schedule  baselines 

-  Cost  overruns 

-  Poor  analysis,  planning,  and  execution 

-  Considerable  program  risk 

-  Unhappy  sponsors/users 
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Goals  of  This  Research 
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■  Provide  Acquisition  Executives 

-  Data  to  validate  schedule  realism 

-  Identify  schedule  trends 

-  Identify  schedule  correlations  to  program  elements 

-  Identify  any  correlations  to  major  policies,  initiatives,  and  laws 

■  Provide  Program  Offices 

-  Data  to  help  shape  realistic 
schedules 

-  Identify  where  their  schedule  is 
an  outlier  to  related  programs 

■  Identify  additional  schedule 
analysis  opportunities  and 
applications 
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Research  Questions 
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■  What  are  the  average  timelines  for  MDAP  and  MAIS  programs? 

■  What  are  the  schedule  trends  for  these  programs? 

■  How  do  schedules  vary  by: 

-  Cost 

-  ACAT 

-  Service 

-  Joint  Capability  Area 

-  %  RDT&E  $  vs  Procurement 

■  Are  there  correlations  to  key  program  attributes? 

■  How  does  the  schedule  data  analysis  align  with  major  policies, 
initiatives,  and  acquisition  reform  legislation? 
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Schedule  Data  Summary 


Programs 
Army 
Navy 
Air  Force 
DoD 

Data  Points 
Schedule  Data  Points 
Avg.  Lifecycle  Cost  (CY15) 


MDAP 

MAIS 

80 

63 

16  (20%) 

16(25%) 

35  (44%) 

13(21%) 

26  (32%) 

15  (24%) 

3(4%) 

19  (30%) 

1,400 

1,250 

287 

274 

$37B 

$2.3B 

■  Collected  data  from  DAMIR  for  all  MDAPs  and  MAIS  Programs 

■  Synthesized,  synchronized,  and  cleansed  data 

■  Analyzed  and  assessed  data  holistically  and  by  segments 
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Data  Characterization 
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Largest  data  sets  for  MS  B,  MS  C,  and  IOC  milestones 
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Methodology 


■  Methodology 


-  Identified  and  reviewed 
primary  data  sources 

-  Developed  list  of  program 
attributes  to  evaluate 

Created  data  framework 

-  Collected  data 

-  Synthesized  and 
synchronized  data 

-  Analyzed  and  assessed  data 

-  Captured  findings 


Regression 

Analysis 


Goodness  of  Fit 


-  Documented  research 
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Average  MDAP  and  MAIS  Timeframes 

80  MDAPs,  63  MAIS  -  1990’s  to  Today 
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MDD 


FRR 


Materiel  Solution 
Analysis  Phase 

Technology  Maturation  and 
Risk  Reduction  Phase 

Engineering  and  Manufacturing 
Development  Phase 

Production  and  Deployment  Phase 

A  B  MDAP  A 

43  Months 

68  Months 

42  Months 

IOC 


12.75  Years 


FDD 
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MDAP  Development  Timelines 

MS  B  — >  MS  C 


Months 


90 

mmm 

Months 

76 

Months 

90's 


OO's 


10's 


<48  Months  48-72  Months  ■  >72  Months 
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MAIS  Development  Timelines 

MS  B  — ►  MS  C 


90's 


OO's 


10's 


<24  Months  24-48  Months  ■  >48  Months 
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MDAP  Schedules  -  Acquisition  Milestones 


MS  A  to  MS  B 

15 

10m 

28m 

41 

57m 

81m 

MS  B  to  MS  C 

41 

11m 

43m 

62 

76m 

234m 

MS  C  to  IOC 

30 

2m 

27m 

38 

46m 

146m 
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MAIS  Schedules  -  Acquisition  Milestones 


MS  A  to  MS  B  9  8m  24m  32m  33m  82m 

MS  B  to  MS  C  31  8m  18m  37m  54m  97m 

MS  C  to  FDD  26  2m  9m  16m  41m  72m 
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Months 


MDAP  Acquisition  Cost  vs.  Schedule 


MDAP 

180 


>$15B  $5B-$15B  <$5B 

■  MS  A -MSB  bMSB-MSC  bMSC-IOC  ■  Total 
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Months 


MAIS  Acquisition  Cost  vs.  Schedule 
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110 
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90 

80 

70 

60 

50 

40 

30 

20 

10 

0 


MAIS 


>$1B  $500M-$1B  <$500M 

■  MS  A -MSB  ■  MS  B  -  MS  C  ■  MS  C  -  FDD  ■  Total 


©  2016  The  MITRE  Corporation.  All  rights  reserved.  Approved  for  Public  Release;  Distribution  Unlimited.  Case  Number  16-0740 


MITRE 


I  15  | 


MDAP  Acquisition  Cost  vs.  Schedule 

MS  B  to  MS  C 
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MAIS  Acquisition  Cost  vs.  Schedule 

MS  B  to  MS  C 


<$500M  13  8m  15m  22m  41m  58m 


$500M-$1B  8  14m  15m  22m  35m  97m 

>$1B  10  23m  37m  64m  89m  91m 
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MDAP  RDT&E  Percentage  vs.  Schedule 

MS  B  to  MS  C 
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MAIS  RDT&E  Percentage  vs.  Schedule 

MS  B  to  MS  C 
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MDAP  RDT&E  Percentage  vs.  Schedule 

MS  B  to  MS  C 


>66%  2  45  48  51  54  57 


33% -66%  11  25  48  67  80  103 

<33%  27  11  46  61  78  234 
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MAIS  RDT&E  Percentage  vs.  Schedule 

MS  B  to  MS  C 


>66%  8  8m  15m  27m  38m  97m 


33%-66%  8  14m  35m  37m  41m  58m 

<33%  12  11m  18m  32m  70m  91m 
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Conclusions 


■  Data  suggests  some  emergent  trends 

-  MS  B  and  MS  C  Durations  decreasing  over  time 

-  Greater  RDT&E  as  a  percentage  of  acquisition  cost  appears  to 
drive  shorter  development  schedules 

■  Data  can  be  leveraged  to  inform  schedule  realism 

■  Data  has  variability  and  a  wide  range  for  schedule  milestones 

■  Complexity  of  DoD  large  scale  programs  not  easily  explained  by 
predictive  parameters  such  as  cost,  type,  service 


■  Recommend  data  be  augmented  if  additional  data  sources  are 
uncovered  and/or  program  data  (actuals)  -  especially  civilian 
and  small/med  DoD  programs 

■  Invitation  to  participate  and  contribute  data  sources 
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Thank  You 


jmanring@mitre.org  804-741-3823 

tfugate@m  itre.org  7  57-758-0563 
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